Towards an Implementation-STakeholder Engagement Model (I-STEM) for improving health and social care services.

BACKGROUND: The implementation science literature acknowledges a need for engagement of key stakeholders when designing, delivering and evaluating implementation work. To date, the literature reports minimal or focused stakeholder engagement, where stakeholders are engaged in either barrier identification and/or barrier prioritisation. This paper begins to answer calls from the literature for the development of tools and guidance to support comprehensive stakeholder engagement in implementation research and practice. The paper describes the systematic development of the Implementation-STakeholder Engagement Model (I-STEM) in the context of an international, large-scale empirical implementation study (ImpleMentAll) aimed at evaluating the effectiveness of a tailored implementation toolkit. The I-STEM is a sensitising tool that defines key considerations and activities for undertaking stakeholder engagement activities across an implementation process. METHODS: In-depth, semistructured interviews and observations were conducted with implementers who were tailoring implementation strategies to integrate and embed internet-based cognitive behavioural therapy (iCBT) services in 12 routine mental health care organisations in nine countries in Europe and Australia. The analytical process was informed by principles of first- and third-generation Grounded Theory, including constant comparative method. RESULTS: We conducted 55 interviews and observed 19 implementation-related activities (e.g., team meetings and technical support calls). The final outcome of our analysis is expressed in an initial version of the I-STEM, consisting of five interrelated concepts: engagement objectives, stakeholder mapping, engagement approaches, engagement qualities and engagement outcomes. Engagement objectives are goals that implementers plan to achieve by working with stakeholders in the implementation process. Stakeholder mapping involves identifying a range of organisations, groups or people who may be instrumental in achieving the engagement objectives. Engagement approaches define the type of work that is undertaken with stakeholders to achieve the engagement objectives. Engagement qualities define the logistics of the engagement approach. Lastly, every engagement activity may result in a range of engagement outcomes. CONCLUSION: The I-STEM represents potential avenues for substantial stakeholder engagement activity across key phases of an implementation process. It provides a conceptual model for the planning, delivery, evaluation and reporting of stakeholder engagement activities. The I-STEM is nonprescriptive and highlights the importance of a flexible, iterative approach to stakeholder engagement. It is developmental and will require application and validation across a range of implementation activities. PATIENT OR PUBLIC CONTRIBUTION: Patient contribution to ImpleMentAll trial was facilitated by GAMIAN-Europe at all stages-from grant development to dissemination. GAMIAN-Europe brings together a wide variety of patient representation organisations (local, regional and national) from almost all European countries. GAMIAN-Europe was involved in pilot testing the ItFits-toolkit and provided their views on the various aspects, including stakeholder engagement. Patients were also represented in the external advisory board providing support and advice on the design, conduct and interpretation of the wider project, including the development of the ItFits-toolkit. TRIAL REGISTRATION: ClinicalTrials.gov NCT03652883. Retrospectively registered on 29 August 2018.

Implementing a revised online screening tool in a routine care online clinic treating anxiety and depression.

Introduction: The ItFits implementation toolkit was developed as part of the ImpleMentAll EU Project, to help guide implementation processes. The ItFits toolkit was tested in the online clinic, Internetpsykiatrien, in the Region of Southern Denmark, where it was employed to optimize screening and intake procedures. We hypothesized that a larger proportion of assessed patients would be referred to treatment. Further, we hypothesized the completion rate and effectiveness would increase, as a result of including a more relevant sample. Method: Using the ItFits-toolkit, Internetpsykiatrien developed a revised online screening tool. Data on patient flow and symptom questionnaires was extracted from Internetpsykiatrien six months prior to- and six months after implementation of the revised online screening tool. Results: A total of 1,830 applicants self-referred for treatment during the study period. A significantly lower proportion of patients were referred to treatment after implementation of the revised screening tool (pre-implementation, n = 1,009; post-implementation, n = 821; odds ratio 0.67, 95% CI: 0.51; 0.87). The number of patients that completed treatment increased significantly (pre-implementation: 136/275 [49.45%], post-implementation, n = 102/162 [62.96%]; odds ratio 1.79, 95% CI 1.20; 2.70). The treatment effect was unchanged (B = 0.01, p = .996). Worth noting, the number of patients that canceled their appointment for the video assessment interview decreased drastically. Conclusion: By using the ItFits toolkit for a focused and structured implementation effort, the clinic was able to improve the completion rate, which is an important effect in iCBT. However, contrary to our hypotheses, we did not find an increase in clinical effect, nor a larger ratio being referred to treatment after assessment. The decreased number of referrals for treatment could be a result of increased awareness of inclusion criteria among the clinicians.

Therapist perceptions of the implementation of a new screening procedure using the ItFits-toolkit in an iCBT routine care clinic: A mixed-methods study using the consolidated framework for implementation research.

Introduction: This study investigates the implementation of a new, more automated screening procedure using the ItFits-toolkit in the online clinic, Internet Psychiatry (iPsych) (www.internetpsykiatrien.dk), delivering guided iCBT for mild to moderate anxiety and depressive disorders. The study focuses on how the therapists experienced the process. Methods: Qualitative data were collected from semi-structured individual interviews with seven therapists from iPsych. The interviews were conducted using an interview guide with questions based on the Consolidated Framework for Implementation Research (CFIR). Quantitative data on the perceived level of normalization were collected from iPsych therapists, administrative staff, and off-site professionals in contact with the target demographic at 10-time points throughout the implementation. Results: The therapists experienced an improvement in the intake procedure. They reported having more relevant information about the patients to be used during the assessment and the treatment; they liked the new design better; there was a better alignment of expectations between patients and therapists; the patient group was generally a better fit for treatment after implementation; and more of the assessed patients were included in the program. The quantitative data support the interview data and describe a process of normalization that increases over time. Discussion: The ItFits-toolkit appears to have been an effective mediator of the implementation process. The therapists were aided in the process of change, resulting in an enhanced ability to target the patients who can benefit from the treatment program, less expenditure of time on the wrong population, and more satisfied therapists.

Effectiveness of Self-guided Tailored Implementation Strategies in Integrating and Embedding Internet-Based Cognitive Behavioral Therapy in Routine Mental Health Care: Results of a Multicenter Stepped-Wedge Cluster Randomized Trial.

BACKGROUND: Internet-based cognitive behavioral therapy (iCBT) services for common mental health disorders have been found to be effective. There is a need for strategies that improve implementation in routine practice. One-size-fits-all strategies are likely to be ineffective. Tailored implementation is considered as a promising approach. The self-guided integrated theory-based Framework for intervention tailoring strategies toolkit (ItFits-toolkit) supports local implementers in developing tailored implementation strategies. Tailoring involves identifying local barriers; matching selected barriers to implementation strategies; developing an actionable work plan; and applying, monitoring, and adapting where necessary. OBJECTIVE: This study aimed to compare the effectiveness of the ItFits-toolkit with implementation-as-usual (IAU) in implementing iCBT services in 12 routine mental health care organizations in 9 countries in Europe and Australia. METHODS: A stepped-wedge cluster randomized trial design with repeated measures was applied. The trial period lasted 30 months. The primary outcome was the normalization of iCBT delivery by service providers (therapists, referrers, IT developers, and administrators), which was measured with the Normalization Measure Development as a proxy for implementation success. A 3-level linear mixed-effects modeling was applied to estimate the effects. iCBT service uptake (referral and treatment completion rates) and implementation effort (hours) were used as secondary outcomes. The perceived satisfaction (Client Satisfaction Questionnaire), usability (System Usability Scale), and impact of the ItFits-toolkit by implementers were used to assess the acceptability of the ItFits-toolkit. RESULTS: In total, 456 mental health service providers were included in this study. Compared with IAU, the ItFits-toolkit had a small positive statistically significant effect on normalization levels in service providers (mean 0.09, SD 0.04; P=.02; Cohen d=0.12). The uptake of iCBT by patients was similar to that of IAU. Implementers did not spend more time on implementation work when using the ItFits-toolkit and generally regarded the ItFits-toolkit as usable and were satisfied with it. CONCLUSIONS: The ItFits-toolkit performed better than the usual implementation activities in implementing iCBT services in routine practice. There is practical utility in the ItFits-toolkit for supporting implementers in developing and applying effective tailored implementation strategies. However, the effect on normalization levels among mental health service providers was small. These findings warrant modesty regarding the effectiveness of self-guided tailored implementation of iCBT services in routine practice. TRIAL REGISTRATION: ClinicalTrials.gov NCT03652883; https://clinicaltrials.gov/ct2/show/NCT03652883. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.1186/s13063-020-04686-4.

Organisational implementation climate in implementing internet-based cognitive behaviour therapy for depression.

BACKGROUND: Internet-based Cognitive Behaviour Therapy (iCBT) for depression have been implemented in routine care across Europe in varying ways, at various scales and with varying success. This study aimed to advance our understanding of organisational implementation climate from the perspectives of implementers and mental health service deliverers. METHODS: Qualitative and quantitative methods were combined to study the concept of organisational implementation climate in mental health care settings. Based on concept mapping, a qualitative workshop with implementers was used to conceptualise organisational implementation climate for optimizing iCBT use in routine practice. Service deliverers involved in the provision of iCBT were invited to participate in an explorative cross-sectional survey assessing levels of satisfaction and usability of iCBT, and organisational implementation climate in implementing iCBT. The two methods were applied independently to study viewpoints of implementers as well as service deliverers. Corresponding to the explorative nature of the study, inductive reasoning was applied to identify patterns and develop a reasonable explanation of the observations made. Correlative associations between satisfaction, usability and implementation climate were explored. RESULTS: Sixteen implementers representing fourteen service delivery organisations across Europe participated in the workshop. The top-three characteristics of a supportive organisational implementation climate included: (1) clear roles and skills of implementers, (2) feasible implementation targets, and (3) a dedicated implementation team. The top-three tools for creating a supportive implementation climate included: (1) feedback on job performance, (2) progress monitoring in achieving implementation targets, and (3) guidelines for assessing the impact of iCBT. The survey (n=111) indicated that service providers generally regarded their organisational implementation climate as supportive in implementing iCBT services. Organisational implementation climate was weakly associated with perceived usability and moderately with satisfaction with iCBT services. CONCLUSIONS: Organisational implementation climate is a relevant factor to implementers and service deliverers in implementing iCBT in routine care. It is not only an inherent characteristic of the context in which implementation takes place, it can also be shaped to improve implementation of iCBT services. Future research should further theorise organisational implementation climate and empirically validate the measurement instruments such as used in this study.

Evaluation of Normalization After Implementation of the Digital Dutch Obstetric Telephone Triage System: Mixed Methods Study With a Questionnaire Survey and Focus Group Discussion.

BACKGROUND: The Dutch Obstetric Telephone Triage System (DOTTS) was developed to improve the quality of acute obstetric care. To achieve optimal effect, the DOTTS should be adopted in the daily care process by triage staff. OBJECTIVE: The primary aim was to evaluate the degree of implementation (ie, normalization) of the DOTTS, and the secondary aim was to evaluate which lessons can be learned from its current implementation in Dutch hospitals. METHODS: An evaluation study with a mixed methods design was performed. All triage staff in 9 Dutch hospitals that implemented the DOTTS before September 1, 2019, were invited to complete the Normalization Measure Development (NoMAD) questionnaire between December 2019 and July 2020. The questionnaire is based on the Normalization Process Theory (NPT). This self-reported questionnaire provides insights into the work people do in order to integrate and embed new practice in routine care. The NPT is based on the following 4 constructs: coherence, cognitive participation, collective action, and reflexive monitoring. Within the questionnaire, each construct is represented by 4-7 questions. Questions are scored on a 5-point normalization process scale. Descriptive statistics were used for analysis of questionnaire scores. Subsequently, the questionnaire findings were discussed during a focus group. Template analysis following the 4 constructs was used for analyzing the results of the focus group. RESULTS: Overall, 173 of 294 (58.8%) triage staff members completed the NoMAD questionnaire, and 90.2% (156/173) of the participants had used the DOTTS for over 6 months. The digital application was used as much as possible or always by 137 of 173 (79.2%) participants. The overall normalization process score was 3.77 (SD 0.36). The constructs coherence, cognitive participation, collective action, and reflexive monitoring scored 4.01 (SD 0.47), 4.05 (SD 0.45), 3.5 (SD 0.45), and 3.72 (SD 0.47), respectively. Analysis of the focus group discussion showed that the added value of the DOTTS was seen as a quality improvement for the care of pregnant women. Dedication of the complete multidisciplinary implementation team was important for facilitating normalization. Support from the medical staff and proper use by all disciplines involved in the triage were seen as facilitating factors. Participants appreciated training and evaluation, and indicated a need for ongoing training and evaluation in relation to goal achievement. CONCLUSIONS: The DOTTS has been integrated into normal care in daily practice. Evaluation by the NoMAD questionnaire provided a positive overall score. These results are in line with or, in some aspects, better than the results of other evaluation studies. Key factors in the normalization process of the DOTTS in obstetric triage are the shared added value for stakeholders, the dedication of the complete multidisciplinary implementation team, and implementation plans that are tailor made in the practical context of the hospital.

The Effect of Dissemination Pathways on Uptake and Relative Costs for a Transdiagnostic, Self-guided Internet Intervention for Reducing Depression, Anxiety, and Suicidal Ideation: Comparative Implementation Study.

BACKGROUND: Self-guided web-based programs are effective; however, inadequate implementation of these programs limits their potential to provide effective and low-cost treatment for common mental health problems at scale. There is a lack of research examining optimal methods for the dissemination of web-based programs in the community. OBJECTIVE: This study aimed to compare the uptake, reach, relative costs, and adherence associated with 3 community-based pathways for delivering a low-intensity web-based transdiagnostic mental health program. The 3 dissemination pathways were social media advertising, advertising in general practice, and advertising in pharmacies. METHODS: Participants were recruited on the web, from general practices, or from community pharmacies; completed a screener for psychological distress; and were offered the 4-week FitMindKit program-a 12-module psychotherapeutic intervention. Uptake was defined as the number of participants who enrolled in the web-based program; reach was defined as the rate of uptake per exposure; and costs were calculated based on staff time, equipment, and advertising. Adherence was assessed as the number of modules of FitMindKit completed by the participants. RESULTS: Uptake comprised 1014 participants who were recruited through the 3 dissemination pathways: on the web (991/1014, 97.73%), in general practice (16/1014, 1.58%), and in pharmacy (7/1014, 0.69%). Reach was highest for social media: 1 in every 50 people exposed to web-based advertising took up the intervention compared with 1 in every 441 in general practitioner clinics and 1 in every 1708 in pharmacies. The dissemination cost was US $4.87 per user on social media, US $557 per user for general practitioner clinics, and US $1272 per user for pharmacy dissemination. No significant differences in adherence were observed between the conditions, whereas all pathways showed an underrepresentation of men and linguistic diversity. CONCLUSIONS: The web-based dissemination pathway was the most efficient and cost-effective for delivering a self-guided internet-based mental health program to people in the community. More research is needed to identify how best to engage men and those with culturally diverse backgrounds in web-based interventions. TRIAL REGISTRATION: Australian New Zealand Clinical Trials Registry ACTRN12618001688279; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=376113.

Organizational Readiness for Implementing an Internet-Based Cognitive Behavioral Therapy Intervention for Depression Across Community Mental Health Services in Albania and Kosovo: Directed Qualitative Content Analysis.

BACKGROUND: The use of digital mental health programs such as internet-based cognitive behavioral therapy (iCBT) holds promise in increasing the quality and access of mental health services. However very little research has been conducted in understanding the feasibility of implementing iCBT in Eastern Europe. OBJECTIVE: The aim of this study was to qualitatively assess organizational readiness for implementing iCBT for depression within community mental health centers (CMHCs) across Albania and Kosovo. METHODS: We used qualitative semistructured focus group discussions that were guided by Bryan Weiner's model of organizational readiness for implementing change. The questions broadly explored shared determination to implement change (change commitment) and shared belief in their collective capability to do so (change efficacy). Data were collected between November and December 2017. A range of health care professionals working in and in association with CMHCs were recruited from 3 CMHCs in Albania and 4 CMHCs in Kosovo, which were participating in a large multinational trial on the implementation of iCBT across 9 countries (Horizon 2020 ImpleMentAll project). Data were analyzed using a directed approach to qualitative content analysis, which used a combination of both inductive and deductive approaches. RESULTS: Six focus group discussions involving 69 mental health care professionals were conducted. Participants from Kosovo (36/69, 52%) and Albania (33/69, 48%) were mostly females (48/69, 70%) and nurses (26/69, 38%), with an average age of 41.3 years. A directed qualitative content analysis revealed several barriers and facilitators potentially affecting the implementation of digital CBT interventions for depression in community mental health settings. While commitment for change was high, change efficacy was limited owing to a range of situational factors. Barriers impacting "change efficacy" included lack of clinical fit for iCBT, high stigma affecting help-seeking behaviors, lack of human resources, poor technological infrastructure, and high caseload. Facilitators included having a high interest and capability in receiving training for iCBT. For "change commitment," participants largely expressed welcoming innovation and that iCBT could increase access to treatments for geographically isolated people and reduce the stigma associated with mental health care. CONCLUSIONS: In summary, participants perceived iCBT positively in relation to promoting innovation in mental health care, increasing access to services, and reducing stigma. However, a range of barriers was also highlighted in relation to accessing the target treatment population, a culture of mental health stigma, underdeveloped information and communications technology infrastructure, and limited appropriately trained health care workforce, which reduce organizational readiness for implementing iCBT for depression. Such barriers may be addressed through (1) a public-facing campaign that addresses mental health stigma, (2) service-level adjustments that permit staff with the time, resources, and clinical supervision to deliver iCBT, and (3) establishment of a suitable clinical training curriculum for health care professionals. TRIAL REGISTRATION: ClinicalTrials.gov NCT03652883; https://clinicaltrials.gov/ct2/show/NCT03652883.

The impact of the early phase of the COVID-19 pandemic on mental-health services in Europe.

PURPOSE: The current COVID-19 pandemic confronts psychiatric patients and mental health services with unique and severe challenges. METHODS: In order to identify these trans-national challenges across Europe, an ad-hoc survey was conducted among 23 experts, each answering for one European or aligned country. RESULTS: A number of important themes and issues were raised for the impact of COVID-19 on mental health and mental health services, barriers to service provision and future consequences. A number of key issues were reported by colleagues across several jurisdictions, even though these were at different stages of their national epidemics. CONCLUSIONS: Based on these findings, we articulate some important learnings from the early stages of the COVID-19 European pandemic, and highlight key considerations for all countries' mental health services as the current pandemic develops and for future pandemics.

Tailored implementation of internet-based cognitive behavioural therapy in the multinational context of the ImpleMentAll project: a study protocol for a stepped wedge cluster randomized trial.

BACKGROUND: Internet-based Cognitive Behavioural Therapy (iCBT) is found effective in treating common mental disorders. However, the use of these interventions in routine care is limited. The international ImpleMentAll study is funded by the European Union's Horizon 2020 programme. It is concerned with studying and improving methods for implementing evidence-based iCBT services for common mental disorders in routine mental health care. A digitally accessible implementation toolkit (ItFits-toolkit) will be introduced to mental health care organizations with the aim to facilitate the ongoing implementation of iCBT services within local contexts. This study investigates the effectiveness of the ItFits-toolkit by comparing it to implementation-as-usual activities. METHODS: A stepped wedge cluster randomized controlled trial (SWT) design will be applied. Over a trial period of 30 months, the ItFits-toolkit will be introduced sequentially in twelve routine mental health care organizations in primary and specialist care across nine countries in Europe and Australia. Repeated measures are applied to assess change over time in the outcome variables. The effectiveness of the ItFits-toolkit will be assessed in terms of the degree of normalization of the use of the iCBT services. Several exploratory outcomes including uptake of the iCBT services will be measured to feed the interpretation of the primary outcome. Data will be collected via a centralized data collection system and analysed using generalized linear mixed modelling. A qualitative process evaluation of routine implementation activities and the use of the ItFits-toolkit will be conducted within this study. DISCUSSION: The ImpleMentAll study is a large-scale international research project designed to study the effectiveness of tailored implementation. Using a SWT design that allows to examine change over time, this study will investigate the effect of tailored implementation on the normalization of the use of iCBT services and their uptake. It will provide a better understanding of the process and methods of tailoring implementation strategies. If found effective, the ItFits-toolkit will be made accessible for mental health care service providers, to help them overcome their context-specific implementation challenges. TRIAL REGISTRATION: ClinicalTrials.gov NCT03652883 . Retrospectively registered on 29 August 2018.

Effects of Internet-Based Cognitive Behavioral Therapy in Routine Care for Adults in Treatment for Depression and Anxiety: Systematic Review and Meta-Analysis.

BACKGROUND: Although there is evidence for the efficacy of internet-based cognitive behavioral therapy (iCBT), the generalizability of results to routine care is limited. OBJECTIVE: This study systematically reviews effectiveness studies of guided iCBT interventions for the treatment of depression or anxiety. METHODS: The acceptability (uptake, participants' characteristics, adherence, and satisfaction), effectiveness, and negative effects (deterioration) of nonrandomized pre-post designs conducted under routine care conditions were synthesized using systematic review and meta-analytic approaches. RESULTS: A total of 19 studies including 30 groups were included in the analysis. Despite high heterogeneity, individual effect sizes of investigated studies indicate clinically relevant changes, with effect sizes ranging from Hedges' g=0.42-1.88, with a pooled effect of 1.78 for depression and 0.94 for anxiety studies. Uptake, participants' characteristics, adherence, and satisfaction indicate a moderate to high acceptability of the interventions. The average deterioration across studies was 2.9%. CONCLUSIONS: This study provides evidence supporting the acceptability and effectiveness of guided iCBT for the treatment of depression and anxiety in routine care. Given the high heterogeneity between interventions and contexts, health care providers should select interventions that have been proven in randomized controlled clinical trials. The successful application of iCBT may be an effective way of increasing health care in multiple contexts.

Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis.

BACKGROUND: The System Usability Scale (SUS) is used to measure usability of internet-based Cognitive Behavioural Therapy (iCBT). However, whether the SUS is a valid instrument to measure usability in this context is unclear. The aim of this study is to assess the factor structure of the SUS, measuring usability of iCBT for depression in a sample of professionals. In addition, the psychometric properties (reliability, convergent validity) of the SUS were tested. METHODS: A sample of 242 professionals using iCBT for depression from 6 European countries completed the SUS. Confirmatory Factor Analysis (CFA) was conducted to test whether a one-factor, two-factor, tone-model or bi-direct model would fit the data best. Reliability was assessed using complementary statistical indices (e.g. omega). To assess convergent validity, the SUS total score was correlated with an adapted Client Satisfaction Questionnaire (CSQ-3). RESULTS: CFA supported the one-factor, two-factor and tone-model, but the bi-factor model fitted the data best (Comparative Fit Index = 0.992, Tucker Lewis Index = 0.985, Root Mean Square Error of Approximation = 0.055, Standardized Root Mean Square Residual = 0.042 (respectively χ2diff (9) = 69.82, p < 0.001; χ2diff (8) = 33.04, p < 0.001). Reliability of the SUS was good (ω = 0.91). The total SUS score correlated moderately with the CSQ-3 (CSQ1 rs = .49, p < 0.001; CSQ2 rs = .46, p < 0.001; CSQ3 rs = .38, p < 0.001), indicating convergent validity. CONCLUSIONS: Although the SUS seems to have a multidimensional structure, the best model showed that the total sumscore of the SUS appears to be a valid and interpretable measure to assess the usability of internet-based interventions when used by professionals in mental healthcare.

Health technology assessment frameworks for eHealth: A systematic review.

OBJECTIVES: Traditionally, health technology assessment (HTA) focuses on assessing the impact of pharmaceutical technologies on health and care. Resources are scarce and policy makers aim to achieve effective, accessible health care. eHealth innovations are increasingly more integrated in all healthcare domains. However, how eHealth is assessed prior to its implementation in care practices is unclear. To support evidence-informed policy making, this study aimed to identify frameworks and methods for assessing eHealth's impact on health care. METHODS: The scientific literature in five bibliographical databases was systematically reviewed. Articles were included if the study was conducted in a clinical setting, used an HTA framework and assessed an eHealth service. A systematic qualitative narrative approach was applied for analysis and reporting. RESULTS: Twenty-one HTA frameworks were identified in twenty-three articles. All frameworks addressed outcomes related to the technical performance and functionalities of eHealth service under assessment. The majority also addressed costs (n = 19), clinical outcomes (n = 14), organizational (n = 15) and system level aspects (n = 13). Most frameworks can be classified as dimensional (n = 13), followed by staged (n = 3), hybrid (n = 3), and business modeling frameworks (n = 2). Six frameworks specified assessment outcomes and methods. CONCLUSIONS: HTA frameworks are available for a-priori impact assessment of eHealth services. The frameworks vary in assessment outcomes, methods, and specificity. Demonstrated applicability in practice is limited. Recommendations include standardization of: (i) reporting characteristics of eHealth services, and (ii) specifying assessment outcomes and methods following a stepped-approach tailored to the functional characteristics of eHealth services. Standardization might improve the quality and comparability of eHTA assessments.

Toward an Objective Assessment of Implementation Processes for Innovations in Health Care: Psychometric Evaluation of the Normalization Measure Development (NoMAD) Questionnaire Among Mental Health Care Professionals.

BACKGROUND: Successfully implementing eMental health (eMH) interventions in routine mental health care constitutes a major challenge. Reliable instruments to assess implementation progress are essential. The Normalization MeAsure Development (NoMAD) study developed a brief self-report questionnaire that could be helpful in measuring implementation progress. Based on the Normalization Process Theory, this instrument focuses on 4 generative mechanisms involved in implementation processes: coherence, cognitive participation, collective action, and reflexive monitoring. OBJECTIVE: The aim of this study was to translate the NoMAD questionnaire to Dutch and to confirm the factor structure in Dutch mental health care settings. METHODS: Dutch mental health care professionals involved in eMH implementation were invited to complete the translated NoMAD questionnaire. Confirmatory factor analysis (CFA) was conducted to verify interpretability of scale scores for 3 models: (1) the theoretical 4-factor structure, (2) a unidimensional model, and (3) a hierarchical model. Potential improvements were explored, and correlated scale scores with 3 control questions were used to assess convergent validity. RESULTS: A total of 262 professionals from mental health care settings in the Netherlands completed the questionnaire (female: 81.7%; mean age: 45 [SD=11]). The internal consistency of the 20-item questionnaire was acceptable (.62≤alpha≤.85). The theorized 4-factor model fitted the data slightly better in the CFA than the hierarchical model (Comparative Fit Index=0.90, Tucker Lewis Index=0.88, Root Mean Square Error of Approximation=0.10, Standardized Root Mean Square Residual=0.12, χ22=22.5, P≤.05). However, the difference is small and possibly not outweighing the practical relevance of a total score and subscale scores combined in one hierarchical model. One item was identified as weak (λCA.2=0.10). A moderate-to-strong convergent validity with 3 control questions was found for the Collective Participation scale (.47≤r≤.54, P≤.05). CONCLUSIONS: NoMAD's theoretical factor structure was confirmed in Dutch mental health settings to acceptable standards but with room for improvement. The hierarchical model might prove useful in increasing the practical utility of the NoMAD questionnaire by combining a total score with information on the 4 generative mechanisms. Future research should assess the predictive value and responsiveness over time and elucidate the conceptual interpretability of NoMAD in eMH implementation practices.

Improving Implementation of eMental Health for Mood Disorders in Routine Practice: Systematic Review of Barriers and Facilitating Factors.

BACKGROUND: Electronic mental health interventions (eMental health or eMH) can be used to increase accessibility of mental health services for mood disorders, with indications of comparable clinical outcomes as face-to-face psychotherapy. However, the actual use of eMH in routine mental health care lags behind expectations. Identifying the factors that might promote or inhibit implementation of eMH in routine care may help to overcome this gap between effectiveness studies and routine care. OBJECTIVE: This paper reports the results of a systematic review of the scientific literature identifying those determinants of practices relevant to implementing eMH for mood disorders in routine practice. METHODS: A broad search strategy was developed with high sensitivity to four key terms: implementation, mental health care practice, mood disorder, and eMH. The reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework was applied to guide the review and structure the results. Thematic analysis was applied to identify the most important determinants that facilitate or hinder implementation of eMH in routine practice. RESULTS: A total of 13,147 articles were screened, of which 48 studies were included in the review. Most studies addressed aspects of the reach (n=33) of eMH, followed by intervention adoption (n=19), implementation of eMH (n=6), and maintenance (n=4) of eMH in routine care. More than half of the studies investigated the provision of mental health services through videoconferencing technologies (n=26), followed by Internet-based interventions (n=20). The majority (n=44) of the studies were of a descriptive nature. Across all RE-AIM domains, we identified 37 determinants clustered in six main themes: acceptance, appropriateness, engagement, resources, work processes, and leadership. The determinants of practices are expressed at different levels, including patients, mental health staff, organizations, and health care system level. Depending on the context, these determinants hinder or facilitate successful implementation of eMH. CONCLUSIONS: Of the 37 determinants, three were reported most frequently: (1) the acceptance of eMH concerning expectations and preferences of patients and professionals about receiving and providing eMH in routine care, (2) the appropriateness of eMH in addressing patients' mental health disorders, and (3) the availability, reliability, and interoperability with other existing technologies such as the electronic health records are important factors for mental health care professionals to remain engaged in providing eMH to their patients in routine care. On the basis of the taxonomy of determinants of practices developed in this review, implementation-enhancing interventions can be designed and applied to achieve better implementation outcomes. Suggestions for future research and implementation practice are provided.

Attitudes towards digital treatment for depression: A European stakeholder survey.

BACKGROUND: The integration of digital treatments into national mental health services is on the agenda in the European Union. The E-COMPARED consortium conducted a survey aimed at exploring stakeholders' knowledge, acceptance and expectations of digital treatments for depression, and at identifying factors that might influence their opinions when considering the implementation of these approaches. METHOD: An online survey was conducted in eight European countries: France, Germany, Netherlands, Poland, Spain, Sweden, Switzerland and The United Kingdom. Organisations representing government bodies, care providers, service-users, funding/insurance bodies, technical developers and researchers were invited to participate in the survey. The participating countries and organisations reflect the diversity in health care infrastructures and e-health implementation across Europe. RESULTS: A total of 764 organisations were invited to the survey during the period March-June 2014, with 175 of these organisations participating in our survey. The participating stakeholders reported moderate knowledge of digital treatments and considered cost-effectiveness to be the primary incentive for integration into care services. Low feasibility of delivery within existing care services was considered to be a primary barrier. Digital treatments were regarded more suitable for milder forms of depression. Stakeholders showed greater acceptability towards blended treatment (the integration of face-to-face and internet sessions within the same treatment protocol) compared to standalone internet treatments. Organisations in countries with developed e-health solutions reported greater knowledge and acceptability of digital treatments. CONCLUSION: Mental health stakeholders in Europe are aware of the potential benefits of digital interventions. However, there are variations between countries and stakeholders in terms of level of knowledge about such interventions and their feasibility within routine care services. The high acceptance of blended treatments is an interesting finding that indicates a gradual integration of technology into clinical practice may fit the attitudes and needs of stakeholders. The potential of the blended treatment approach, in terms of enhancing acceptance of digital treatment while retaining the benefit of cost-effectiveness in delivery, should be further explored. FUNDING: The E-COMPARED project has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 603098.

The therapist's role in the implementation of internet-based cognitive behavioural therapy for patients with depression: study protocol.

BACKGROUND: Internet-based Cognitive Behavioural Therapy (iCBT) for the treatment of depressive disorders is innovative and promising. Various studies have demonstrated large effect sizes up to 2.27, but implementation in routine practice lags behind. Mental health therapists play a significant role in the uptake of internet-based interventions. Therefore, it is interesting to study factors that influence the therapists in whether they apply internet-based therapy or not. This study, as part of the European implementation project MasterMind, aims to identity the factors that promote or hinder therapists in the use of iCBT in depression care. METHODS/DESIGN: The uptake of iCBT by therapists in routine mental health care practice for the treatment of depression will be evaluated by a mixed method approach, to provide an understanding of the implementation factors (quantitative), and to ascertain the facilitating and hindering factors in the involvement of therapists in the implementation of iCBT (qualitative). The involvement of therapists in the implementation of iCBT is analysed following the RE-AIM framework on the five dimensions Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. This enables us to evaluate the reach of therapists, the impact of iCBT on depression care, the extent to which therapists adopt iCBT, the extent to which iCBT is delivered as intended, and how iCBT can be maintained over time. DISCUSSION: The results will provide valuable insight into the role of therapists in the implementation of iCBT for depression in secondary mental health care settings. They will result in concrete recommendations for how therapists can be facilitated in implementing and up-scaling iCBT for depression.

Reliability and Validity of Assessing User Satisfaction With Web-Based Health Interventions.

BACKGROUND: The perspective of users should be taken into account in the evaluation of Web-based health interventions. Assessing the users' satisfaction with the intervention they receive could enhance the evidence for the intervention effects. Thus, there is a need for valid and reliable measures to assess satisfaction with Web-based health interventions. OBJECTIVE: The objective of this study was to analyze the reliability, factorial structure, and construct validity of the Client Satisfaction Questionnaire adapted to Internet-based interventions (CSQ-I). METHODS: The psychometric quality of the CSQ-I was analyzed in user samples from 2 separate randomized controlled trials evaluating Web-based health interventions, one from a depression prevention intervention (sample 1, N=174) and the other from a stress management intervention (sample 2, N=111). At first, the underlying measurement model of the CSQ-I was analyzed to determine the internal consistency. The factorial structure of the scale and the measurement invariance across groups were tested by multigroup confirmatory factor analyses. Additionally, the construct validity of the scale was examined by comparing satisfaction scores with the primary clinical outcome. RESULTS: Multigroup confirmatory analyses on the scale yielded a one-factorial structure with a good fit (root-mean-square error of approximation =.09, comparative fit index =.96, standardized root-mean-square residual =.05) that showed partial strong invariance across the 2 samples. The scale showed very good reliability, indicated by McDonald omegas of .95 in sample 1 and .93 in sample 2. Significant correlations with change in depressive symptoms (r=-.35, P<.001) and perceived stress (r=-.48, P<.001) demonstrated the construct validity of the scale. CONCLUSIONS: The proven internal consistency, factorial structure, and construct validity of the CSQ-I indicate a good overall psychometric quality of the measure to assess the user's general satisfaction with Web-based interventions for depression and stress management. Multigroup analyses indicate its robustness across different samples. Thus, the CSQ-I seems to be a suitable measure to consider the user's perspective in the overall evaluation of Web-based health interventions.

European COMPARative Effectiveness research on blended Depression treatment versus treatment-as-usual (E-COMPARED): study protocol for a randomized controlled, non-inferiority trial in eight European countries.

BACKGROUND: Effective, accessible, and affordable depression treatment is of high importance considering the large personal and economic burden of depression. Internet-based treatment is considered a promising clinical and cost-effective alternative to current routine depression treatment strategies such as face-to-face psychotherapy. However, it is not clear whether research findings translate to routine clinical practice such as primary or specialized mental health care. The E-COMPARED project aims to gain knowledge on the clinical and cost-effectiveness of blended depression treatment compared to treatment-as-usual in routine care. METHODS/DESIGN: E-COMPARED will employ a pragmatic, multinational, randomized controlled, non-inferiority trial in eight European countries. Adults diagnosed with major depressive disorder (MDD) will be recruited in primary care (Germany, Poland, Spain, Sweden, and the United Kingdom) or specialized mental health care (France, The Netherlands, and Switzerland). Regular care for depression is compared to "blended" service delivery combining mobile and Internet technologies with face-to-face treatment in one treatment protocol. Participants will be followed up at 3, 6, and 12 months after baseline to determine clinical improvements in symptoms of depression (primary outcome: Patient Health Questionnaire-9), remission of depression, and cost-effectiveness. Main analyses will be conducted on the pooled data from the eight countries (n = 1200 in total, 150 participants in each country). DISCUSSION: The E-COMPARED project will provide mental health care stakeholders with evidence-based information and recommendations on the clinical and cost-effectiveness of blended depression treatment. TRIAL REGISTRATION: France: ClinicalTrials.gov NCT02542891 . Registered on 4 September 2015; Germany: German Clinical Trials Register DRKS00006866 . Registered on 2 December 2014; The Netherlands: Netherlands Trials Register NTR4962 . Registered on 5 January 2015; Poland: ClinicalTrials.Gov NCT02389660 . Registered on 18 February 2015; Spain: ClinicalTrials.gov NCT02361684 . Registered on 8 January 2015; Sweden: ClinicalTrials.gov NCT02449447 . Registered on 30 March 2015; Switzerland: ClinicalTrials.gov NCT02410616 . Registered on 2 April 2015; United Kingdom: ISRCTN registry, ISRCTN12388725 . Registered on 20 March 2015.